Electronic device for recognizing gesture of user from sensor signal of user and method for recognizing gesture using the same

ABSTRACT

Aspects of the disclosure include an electronic device comprising: a sensor device including at least one biometric sensor; memory storing a table of a plurality of gestures, wherein for each gesture, a corresponding plurality of features are stored in the memory; and at least one processor operatively connected to the sensor device and the memory, wherein the at least one processor is configured to: obtain a bio-signal of a user from the at least one biometric sensor; select a section of the bio-signal that includes one feature of the corresponding plurality of features for the plurality of gestures; determine a specific one of the plurality of gestures based on the one feature included in the section of the bio-signal and the corresponding plurality features for the plurality of gestures.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2019-0170152, filed on Dec. 18,2019, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic device capable of recognizing agesture of a user using a sensor signal sensed from the user.

2. Description of Related Art

Electronic devices having touchscreens detect touch inputs to interactwith the display. In addition to touch inputs, touchscreens detectgestures, which involve multiple touch inputs. Moreover, electronicdevices can interpret each of the touch inputs as a single gesture, asopposed to independent touches. Based on the gesture, the electronicdevice can perform different operations.

Electronic devices include both smartphones and wearable devices. It isimportant to provide a user friendly input mechanism for wearabledevices.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure include an electronic device comprising: asensor device including at least one biometric sensor; memory storing atable of a plurality of gestures, wherein for each gesture, acorresponding plurality of features are stored in the memory; and atleast one processor operatively connected to the sensor device and thememory, wherein the at least one processor is configured to: obtain abio-signal of a user from the at least one biometric sensor; select asection of the bio-signal that includes one feature of the correspondingplurality of features for the plurality of gestures; determine aspecific one of the plurality of gestures based on the one featureincluded in the section of the bio-signal and the correspondingplurality features for the plurality of gestures.

Aspects of the disclosure include a method for recognizing a gestureperformed by an electronic device, the method comprising: sensing asensor signal from a user; storing a table of a plurality of gestures,wherein for each gesture, a corresponding plurality of features arestored; setting a crossing point between the sensor signal and a presetreference line and a transition point of the sensor signal; segmentingthe sensor signal using at least one of the crossing point and thetransition point; selecting a section of the sensor signal that includesone feature of the corresponding plurality of features for the pluralityof gestures; and determining a specific one of the plurality of gesturesbased on the one feature included in the segment and the correspondingplurality features for the plurality of gestures.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses certain embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a diagram illustrating an electronic device according to anembodiment;

FIG. 2A and FIG. 2B show diagrams showing forms of extracting a featurevalue from a sensor signal in a recognition method according to acomparative example;

FIG. 3 is a flowchart of a gesture recognition method performed by anelectronic device according to an embodiment;

FIG. 4 is a flowchart of a method for segmenting, by an electronicdevice, a sensor signal using a crossing point;

FIG. 5 is a flowchart of a method for segmenting, by an electronicdevice according to an embodiment, a sensor signal using a transitionpoint;

FIG. 6 is a flowchart of a method for segmenting, by an electronicdevice according to an embodiment, a sensor signal using a crossingpoint and a transition point;

FIG. 7 is a diagram illustrating a form in which a sensor signal issegmented using a crossing point in an embodiment;

FIG. 8 is a diagram illustrating a form in which a sensor signal issegmented using a crossing point and a transition point in anembodiment;

FIG. 9 is a diagram illustrating a form of segmentation of a sensorsignal based on an amount of change in an x value or a y value betweenadjacent transition points in an embodiment;

FIG. 10 is a diagram illustrating a form of segmenting a sensor signalbased on a distance between a transition point and a start point or anend point of a signal section, and selecting a candidate section amongthe segmented sensor signals in an embodiment;

FIG. 11 is a diagram illustrating a form of selecting candidate sectionsbased on an intensity of a sensor signal in an embodiment;

FIG. 12A and FIG. 12B is a diagram illustrating a heartbeat signal of auser through a frequency density function in an embodiment;

FIG. 13A and FIG. 13B illustrate sensor signals sensed when differentusers perform the same gesture; and

FIG. 14 illustrates an electronic device in a network environmentaccording to an embodiment.

With respect to the description of the drawings, the same or similarreference numerals may be used for the same or similar components.

DETAILED DESCRIPTION

Hereinafter, certain embodiments of the disclosure may be described withreference to accompanying drawings. However, this is not intended tolimit the disclosure to a specific embodiment, and is to be understoodas including various modifications, equivalents, and/or alternatives ofthe embodiment of the disclosure.

Among user interface technologies, technology for recognizing gesturesof a user may use an image obtained from an image sensor or an inertialmeasurement unit (IMU) such as an acceleration sensor and a gyro sensor.

In particular, the scheme of recognizing the gesture using the inertialmeasurement unit may have a higher degree of freedom compared to thescheme using the image in that it may be applied to a wearable device.

To recognize the gesture using sensors attached to the wearable device,noise may be removed from continuously sensed sensor signals. Thisallows cutting a gesture section by predicting a start point and an endpoint of the gesture section. Further, time series data of the gesturesection may be used to recognize the gesture.

Time series data can be extracted from a sensor signal and may be inputinto an input layer of a deep learning network, for example, a neuralnetwork, a feature value for the corresponding time series data may beextracted from a hidden layer (e.g., a convolution layer in the deeplearning network), and a gesture may be distinguished in an outputlayer.

However, when the number of hidden layers is not large, it is not verycomplicated in terms of an amount of operation. However, when the numberof hidden layers is increased for more accurate gesture recognition, asthe amount of operation for gesture recognition increases, a high-levelperformance may be required for an electronic device that performs thegesture recognition method.

Therefore, the foregoing may be difficult to be applied in anenvironment, such as a wearable device, since wearable devices have arelatively low operation ability compared to a smartphone, a tablet PC,a desktop, and the like and using a limited battery.

In addition, when using the deep learning network, because it isimpossible to accurately analyze which work was performed inside thedeep learning network, there may be difficulties in commercializing aproduct in that there is no choice but to predict which work wasperformed.

Certain embodiments disclosed in the disclosure may provide anelectronic device that recognizes a gesture of a user from a sensorsignal such that even when the wearable device has lower batterycapacity and limited operations by a processor, the electronic devicemay accurately recognize the gesture performed by the user.

FIG. 1 is a diagram illustrating an electronic device 10 (e.g., anelectronic device 1401 in FIG. 14) according to an embodiment.

Referring to FIG. 1, the electronic device 10 according to an embodimentmay include a sensor device 110 (e.g., a sensor module 1476 or a sensorcircuit in FIG. 14), a processor 120 (e.g., a processor 1420 in FIG.14), and a memory 130 (e.g., a memory 1430 in FIG. 14). The electronicdevice 10 according to the embodiment may be a wearable device, but maynot be limited thereto, and may be a portable terminal such as a smartphone, a tablet, or the like, a patch, a sticker type device, or animplantable device. The term “processor” shall be understood to refer tothe singular and plural contexts.

In an embodiment, the sensor device 110 may be disposed in at least aportion of a housing of the electronic device 10 (e.g., a housing of thewearable device), and may sense a user input. A sensor signal is anelectrical signal, that is generally expressed as either theinstantaneous current or voltage value (now referred to as value) atparticular times. A reference line is a preset value. As a function oftime, the preset value appears as a horizontal line, thus a presetreference line. A crossing point is the instant in time when value ofthe sensor signal changes from below the preset reference line to abovethe preset reference line and vice versa. A transition point is when theslope of the sensor signal changes from positive to negative, or viceversa. The sensor device 110 may set a crossing point between the sensorsignal and a preset reference line and a transition point of the sensorsignal in the sensor signal sensed from the user. The sensor device 110may then segment the sensor signal using at least one of the crossingpoint and the transition point. To this end, the sensor device 110 mayinclude a sensor 111, a signal processor 112 (or a signal processingcircuit), and a sensor hub 113 (or a hardware processor for controllingthe sensor).

In an embodiment, the sensor 111 may sense the sensor signal (e.g., anacceleration signal, an angular velocity signal, a geomagnetic signal, aphotoplethysmography (PPG) signal, and the like) from the user. Forexample, the sensor 111 may include at least one of an accelerationsensor, a PPG sensor, a gyro sensor, a geomagnetic sensor, and anelectrocardiogram sensor.

In an embodiment, the signal processor 112 may process the sensor signalsensed from the sensor 111. For example, the signal processor 112 mayperform a pre-processing operation, a compensation operation, avalidation determination operation, an operation of setting the crossingpoint and the transition point, and a sensor signal segmentationoperation based on the crossing point and the transition point of thesensor signal.

In certain embodiments, the signal processor 112 may perform thepre-processing operation of the sensor signal to perform filtering ofthe sensor signal (e.g., sensor signal filtering using a bandpassfilter). For example, in the process of sensing the sensor signal of theuser, the signal processor 112 may remove a noise signal generated by anoperation not intended by the user other than a gesture operation andfilter only a region of the signal corresponding to a gesture section ofthe user, and may compensate for a sampling rate when necessary.

In an embodiment, the signal processor 112 may perform the compensationoperation of the sensor signal to compensate for a signal section (e.g.,a section in which an intensity of the signal rises to be equal to orabove a specific value or falls to be equal to or below a specific valueduring a unit time) requiring the compensation based on an operation anda posture of the user. For example, the signal processor 112 may performcompensation of deleting the section requiring the compensation of thesensor signal.

In certain embodiments, the signal processor 112 may perform thevalidation determination operation to select a signal valid for gesturerecognition among a plurality of sensor signals sensed from a pluralityof sensors. For example, the signal processor 112 may determine a sensorsignal whose validation determined using at least one of a signal tonoise ratio (SNR), pattern analysis of the sensor signal, and comparisonbetween the sensor signals is equal to or above a preset reference valueas the signal valid for the gesture recognition.

In an embodiment, the signal processor 112 breaks the continuously inputsensor signals into signals of a finite length (sections) through thesensor signal segmentation operation. The signal processor 112 maypredict a section that is likely to contain a gesture, and distinguishmeaningful signals (e.g., sensor signals including gesture signals) andmeaningless signals (e.g., sensor signals not including the gesturesignals) from each other, thereby extracting only the meaningful signals(e.g., signals of portions in which the gesture operation is estimatedto exist).

In an embodiment, the signal processor 112 may perform the operation ofsetting the crossing point and the transition point on the sensor signalincluding the extracted gesture signal to set the crossing point and thetransition point in the sensor signal, and may perform the signalsegmentation operation to segment the sensor signal using at least oneof the crossing point and the transition point.

In an embodiment, the sensor hub 113 may transmit the sensor signal(e.g., the sensor signal segmented based on the sensor signalsegmentation operation) processed by the signal processor 112 to theprocessor 120. In an embodiment, the sensor hub 113 might not transmit asensor signal determined to be invalid based on the validationdetermination operation of the signal processor 112 to the processor120. In an embodiment, when there are a plurality of sensor signalsprocessed through the signal processor 112, the sensor hub 113 maytransmit the plurality of sensor signals to the processor 120sequentially or simultaneously.

In an embodiment, the processor 120 may extract a feature valueindicating the gesture of the user using the sensor signal (e.g., asensor signal whose validation is equal to or above a reference amongthe segmented sensor signals) transmitted from the sensor hub 113. Theprocessor 120 may recognize the gesture of the user using the extractedfeature value.

In an embodiment, the processor 120 may perform all operations (e.g.,the pre-processing operation, the compensation operation, the validationdetermination operation, the operation of setting the crossing point andthe transition point, the sensor signal segmentation operation based onthe crossing point and the transition point, the feature valueextraction operation, the gesture recognition operation, and the like ofthe sensor signal) performed for the gesture recognition after receivingthe sensor signal from the sensor device 110.

In an embodiment, the memory 130 may store various commands or dataassociated with a control operation of the electronic device 10. Thememory 130 may include at least one of a volatile memory and anon-volatile memory, but may not be limited thereto. In an embodiment,the memory 130 may store a gesture set by matching a plurality ofgesture operations with functions respectively corresponding to theplurality of gestures, and may store a list (e.g., Table 1) of featurevalues for each gesture respectively corresponding to types of aplurality of sensors. In an embodiment, the memory 130 may store agesture classification model that outputs a gesture corresponding to aspecific feature value by taking the specific feature value as an inputvalue.

TABLE 1 Maximum Ratio of value of maximum Fast value and Fourier minimumvalue Transform Integral Skewness of sensor signal coefficient valuevalue (Min max ratio Gesture (FFT Max) (Integral) (Skewness) (max/min))Drop 1869 1.2429e+04 −0.1691 22.6 operation (DROP) Lift 300 1.3002e+050.1837 0.06 operation (LIFT) Flick 1.263e+05 1.4265e+07 0.3328 1.402operation (FLICK)

Referring to Table 1, the electronic device 10 may store the list of thefeature values for each gesture as shown in Table 1. When the featurevalue is extracted, the electronic device 10 may load the previouslystored list of the feature values for each gesture from the memory 130and select the gesture corresponding to the feature value using theloaded list of the feature values for each gesture, thereby recognizingthe gesture. For example, when the extracted feature value is a skewnessvalue (Skewness) and the skewness value is 1.3002e+05, the electronicdevice 10 may recognize that the corresponding feature value is a valueindicating a lift operation gesture.

FIG. 2A and FIG. 2B shows diagrams showing forms of extracting a featurevalue from a sensor signal in a gesture recognition method, and FIG. 3is a flowchart of a gesture recognition method performed by theelectronic device 10 (e.g., the electronic device 10 in FIG. 1 or theelectronic device 1401 in FIG. 14) according to an embodiment.

Referring to FIG. 2A, the electronic device may extract the featurevalue from the PPG signal sensed from the user using a specific gesturerecognition method (e.g., a method using an integral value of anentirety of the sensor signal as the feature value). For example, theprocessor of the electronic device may extract the integral valuecalculated by integrating the entirety of the gesture signal as thefeature value. In this connection, because an integral value of an area{circle around (1)} has a positive value and an integral value of anarea {circle around (2)} has a negative value, the integral valuedecreases as the integral value of the area {circle around (1)} and theintegral value of the area {circle around (2)} are added. For example,when the sensor signal exhibits a form of changing from the positivevalue to the negative value or exhibits a form of changing from thenegative value to the positive value, the feature value for recognizingthe gesture may not be able to be accurately extracted. This is becausethe feature value changes by summing feature values of respectivesections. For example, when a degree of change of the integral value ofthe sensor signal from the positive value to the negative value or fromthe negative value to the positive value is large, a result valuederived in the above scheme may distort an original intention or acharacteristic value might not be obtained.

Referring to FIG. 2B, in a case of an acceleration signal in which thenumber of times the sensor signal changes from the positive value to thenegative value or from the negative value to the positive value for thesame gesture operation is greater than that of the PPG signal, positivesections (e.g., areas {circle around (1)}, {circle around (3)}, and{circle around (5)}) and negative sections (e.g., areas {circle around(2)}, {circle around (4)}, and {circle around (6)}) for which theintegral value is to be calculated increase, so that an amount ofoperation for extracting one feature value (e.g., the integral value)increases. Further, as the sum of the areas of the respective sectionsis calculated as the feature value for the gesture, an intention tocalculate a frequently changing value as the feature value may not beable to be accurately reflected, so that the feature value indicatingthe gesture may not be accurately extracted.

Examples of feature values extracted from each area and feature valuesextracted from a total area by applying a gesture recognition scheme forvarious feature values extracted from the PPG signal in FIG. 2A are asshown in a Table below.

TABLE 2 Total area (areas {circle around (1)} Area {circle around (1)}Area {circle around (2)} and {circle around (2)}) Integral value1.4265e+05 −1.3002e+05 1.2429e+04 (Integral) Skewness value −0.16910.1837 0.3328 (Skewness) Kurtosis value 1.7490 1.6129 1.7555 (kurtosis)Fast Fourier 869 729 1.263e+03 Transform maximum value (FFT max value)Amount of 163 179 342 change in X value (X-axis difference) Ratio of22.6 0.06 −1.402 maximum value and minimum value of sensor signal (Minmax ratio (max/min))

Referring to Table 2 above, even in a case of extracting another featurevalue (e.g., the skewness value, the kurtosis value, the Fast Fouriertransform maximum value, the amount of change in the x value, and theratio of the minimum value and the maximum value) extracted from the PPGsignal in FIG. 2A using the recognition method according to thecomparative example, it may not be possible to extract the accuratefeature value because the feature value obtained from the total area(the areas {circle around (1)} and {circle around (2)}) is moredifficult to exhibit a feature of the gesture than the feature valueobtained from each of the areas {circle around (1)} and {circle around(2)}. In particular, when the skewness value is used, a degree ofasymmetry significantly changes depending on which signal section is setas a reference. Because the foregoing gesture recognition method usesthe skewness value for the total area, the value may be distorted or anunintended result may be derived in a waveform in which the number ofchanges between the negative section and the positive section is large.Therefore, subdividing each section and considering each subdividedsection individually in the process of extracting the feature valueindicating the gesture from the sensor signal, may more accuratelyrecognize the gesture.

The electronic device 10 according to an embodiment may elaboratelysegment the sensor signal sensed from the user using at least one of thecrossing point and the transition point, extract feature values from thesegmented sensor signals, respectively, and use the extracted featurevalues to recognize the gesture performed by the user.

Referring to FIG. 3, according to an embodiment, in operation 310, thesensor device 110 may sense the sensor signal from the user. In anembodiment, the sensor device 110 may include the plurality of sensorsof the different types, and may sense the plurality of sensor signals(e.g., the acceleration signal, the PPG signal, a gyro signal, thegeomagnetic signal, and an electrocardiogram signal) using the pluralityof sensors (e.g., the acceleration sensor, the PPG sensor, the gyrosensor, the geomagnetic sensor, and the electrocardiogram sensor).

According to an embodiment, in operation 320, the sensor device 110 mayset or determine the crossing point between the sensor signal sensed inoperation 310 and the preset reference line and the transition point ofthe sensor signal. The reference line may mean a virtual line connectingpreset values with each other. For example, when a value of a bio-signalis a raw value, the reference line may mean a line connecting pointswhose raw values become 0 with each other. Alternatively, when the valueof the bio-signal is not the raw value, the specified reference line maymean a line connecting points where values of the bio-signal normalizedby the processor become 0 with each other. For example, when the valueof the bio-signal is a PPG value, the reference line may mean a lineconnecting points where values obtained by standardizing or normalizingthe PPG value by applying the bandpass filter, a moving average filter,and the like or applying other operations become 0 with each other.

The crossing point may mean a point at which the sensed sensor signalhas a preset value. In an embodiment, when the value of the presetreference line (e.g., the value of the sensor) is 0, the sensor device110 may set a point (e.g., a zero crossing point) at which the referenceline with the sensor value of 0 crosses the sensor signal as thecrossing point. In an embodiment, the sensor device 110 may sense achange in a slope of the sensor signal, and set a point at which theslope changes from a positive value to a negative value or a point atwhich the slope changes from the negative value to the positive value asthe transition point. For example, the sensor device 110 may sensechanges in the values of the sensor signal to set a point at which thevalue of the sensor signal decreases after increasing or a point atwhich the value of the sensor signal increases after decreasing as thetransition point.

According to an embodiment, in operation 330, the sensor device 110 maysegment the sensor signal into a plurality of signal sections using atleast one of the crossing point and the transition point set inoperation 320. In an embodiment, the sensor device 110 may segment thesensor signal using only the crossing point, or segment the sensorsignal using only the transition point. In an embodiment, the sensordevice 110 may primarily segment the sensor signal using the crossingpoint, and secondarily segment the sensor signal that is primarilysegmented using the transition point.

According to an embodiment, in operation 340, the sensor device 110 maytransmit at least one signal section generated in operation 330 to theprocessor 120. In an embodiment, the sensor device 110 may select onlysensor signals determined to be valid signals through the signalprocessor 112 among the sensor signals sensed from the sensor 111, andtransmit the sensor signals determined to be valid signals to theprocessor 120 through the sensor hub 113. In an embodiment, the sensordevice 110 may transmit all the signals sensed from the sensor 111 tothe processor 120 through the sensor hub 113.

According to an embodiment, in operation 350, the processor 120 mayextract the feature value indicating the gesture from each of theplurality of signal sections transmitted from the sensor device 110. Forexample, the processor 120 may extract, from each of the plurality ofsignal sections, at least one of energy, correlation, entropy, a FastFourier Transform coefficient, mean, variance, covariance, a maximumvalue, a minimum value, a zero crossing point, a length of time seriesdata, skewness, kurtosis, and integral values, autocorrelation, acontinuous wavelet transform coefficient, a peak (a local maximum valueand a local minimum value), the number of peaks, and entropy of thesensor signal as the feature value.

In an embodiment, the processor 120 extracts the feature values from theplurality of signal sections. The processor 120 may independentlyextract the feature value from each of the plurality of signal sections.In an embodiment, the processor 120 may define a form of each of theplurality of signal sections using a transition point included in eachof the plurality of signal sections, set a sequence of the plurality ofsignal sections based on the form of each of the plurality of signalsections, and extract the set sequence of the plurality of signalsections as the feature value indicating the gesture of the user. Forexample, the processor 120 may determine whether each of the pluralityof signal sections is concave or convex using the transition pointincluded in each of the plurality of signal sections. The processor 120may define the sequence of the plurality of signal sections in a binaryform by setting a concave signal section as “1” and a convex signalsection as “0”, and may extract the sequence of the plurality of signalsections defined in the binary form as the feature value indicating thegesture.

According to an embodiment, in operation 360, the processor 120 mayrecognize the gesture of the user using the feature values extracted inoperation 350. For example, the processor 120 may input the featurevalues extracted in operation 350 into a gesture classification modelloaded from the memory 150, select gesture data corresponding to aresult value output from the gesture classification model based on thefeature value from a preset gesture set to recognize the gesture of theuser, and recognize the gesture using the selected gesture data. Theelectronic device 10 according to an embodiment may subdivide the sensorsignal sensed from the user using at least one of the crossing point andthe transition point, and extract the feature value indicating thegesture of the user from each of the subdivided signal sections.

In an embodiment, the gesture classification model may be a gestureclassification model (e.g., a classifier of a deep learning model) thathas previously learned sensor signal information (e.g., an amount ofchange, a waveform, and the like of the sensor signal) generated as theuser performs the gesture, and the gestures respectively correspondingto the sensor signals as learning data. The electronic device 10 maystore the gesture classification model in the memory 130, and may updatethe gesture classification model as the user learns the gesture.

A section of a sensor signal shall be understood to mean the collectionof values as a function of time of the sensor signal during a timeinterval.

In FIG. 4, crossing points are used to determine segments of the sensorsignal. However, each segment can only have one transition point. If asegment has more than one transition point, the sensor device resets thereference line.

FIG. 4 is a flowchart of a method for segmenting, by the electronicdevice 10 (the electronic device 10 in FIG. 1 or the electronic device1401 in FIG. 14), a sensor signal using a crossing point.

In an embodiment, the electronic device 10 may segment the sensor signalusing the crossing point. Referring to FIG. 4, according to anembodiment, in operation 410, the sensor device 110 may set a value ofthe reference line for setting the crossing point in the sensor signal.In an embodiment, the sensor device 110 may set the value of thereference line such that the number of crossing points between thesensor signal and the reference line is equal to or greater than apreset number. In an embodiment, the sensor device 110 segments thesensor signal into the plurality of signal sections using the crossingpoint. The sensor device 110 may set the value of the preset referenceline such that at least one transition point is included in each of theplurality of signal sections.

According to an embodiment, in operation 420, the sensor device 110 mayset the point where the sensor signal crosses the reference line set inoperation 410 as the crossing point, and segment the sensor signal intothe plurality of signal sections based on the set crossing point. Forexample, when there are three points (e.g., a first crossing point, asecond crossing point, and a third crossing point) where the sensorsignal crosses the reference line set in operation 410, the sensordevice 110 may segment the sensor signal into a first signal sectionbetween the first crossing point and the second crossing point, and asecond signal section between the second crossing point and the thirdcrossing point.

According to an embodiment, in operation 430, the sensor device 110 maydetermine whether each of the plurality of signal sections generated inoperation 420 includes one transition point. When it is determined thata plurality of transition points are included in one signal section, thesensor device 110 may change the reference line to include onetransition point for each of the plurality of signal sections byperforming operation 410 again. In an embodiment, when the number ofplurality of signal sections is less than the preset number, the sensordevice 110 may re-perform operation 410 to change the value of thereference line for setting the crossing point such that the number ofplurality of signal sections is equal to or greater than the presetnumber.

FIG. 5 is a flowchart of a method for segmenting, by the electronicdevice 10 (e.g., the electronic device 10 in FIG. 1 or the electronicdevice 1401 in FIG. 14) according to an embodiment, a sensor signalusing a transition point.

In an embodiment, the electronic device 10 may segment the sensor signalusing the transition point. Referring to FIG. 5, according to anembodiment, in operation 510, the sensor device 110 may set thetransition point from the sensor signal. For example, the sensor device110 may sense the change in the slope from the sensor signal, and mayset the point at which the slope changes from the positive value to thenegative value or the point at which the slope changes from the negativevalue to the positive value as the transition point. For example, thesensor device 110 may set the point at which the value of the sensorsignal decreases after increasing or the point at which the value of thesensor signal increases after decreasing as the transition point.

In operation 520, the sensor device 110 may segment the sensor signalinto the plurality of signal sections using the transition points set inoperation 510 as a reference. In an embodiment, the sensor device 110may segment the sensor signal such that each of the plurality oftransition points is included in a different signal section, using theplurality of transition points set in operation 510.

In operation 530, the sensor device 110 may determine whether an amountof change in an x value or an amount of change in a y value betweenadjacent transition points among the plurality of transition points isless than a preset amount of change. In an embodiment, the sensor device110 may determine whether both the amount of change in the x value andthe amount of change in the y value between the adjacent transitionpoints are less than the preset amount of change.

When it is determined from operation 530 that the amount of change inthe x value or the amount of change in the y value between the adjacenttransition points are less than the preset amount of change (530-yes),in operation 540, the sensor device 110 may allow two transition pointswhose amount of change in the x value or amount of change in the y valueis less than the preset amount of change to be included in one signalsection. For example, the plurality of transition points may include afirst transition point and a second transition point disposed at aposition adjacent to the first transition point, and the sensor device110 may segment the sensor signal such that the first transition pointand the second transition point are included in different signalsections. In this connection, when an amount of change (an amount ofchange in an x value or an amount of change in a y value) between thefirst transition point and the second transition point is less than thepreset amount of change, the first transition point and the secondtransition point may be included in one signal section. Conversely, whenit is determined that an amount of change in an x value or an amount ofchange in a y value between the adjacent transition points is not lessthan the preset amount of change (530-no), operation 550 may beperformed.

In operation 550, for each of the plurality of signal sections segmentedin operation 540, the sensor device 110 may determine whether a distancebetween each transition point included in each of the plurality ofsignal sections and each start point of each signal section, or adistance between each transition point included in each of the pluralityof signal sections and each end point of each signal section exceeds apreset distance. For example, for the first transition point included inthe first signal section, the sensor device 110 may determine whether adistance between a start point of the first signal section and the firsttransition point or a distance between an end point of the first signalsection and the first transition point exceeds the preset distance.

According to an embodiment, when the distance between each transitionpoint and each start point of each signal section, or the distancebetween each transition point included in each of the plurality ofsignal sections and each end point of each signal section does notexceed the preset distance (550-no), the method may be terminatedwithout executing operation 560. Conversely, when the distance betweeneach transition point and each start point of each signal section, orthe distance between each transition point included in each of theplurality of signal sections and each end point of each signal sectionexceeds the preset distance (550-yes), operation 560 may be executed.According to an embodiment, in operation 560, for each signal section inwhich the distance between each transition point included in each of theplurality of signal sections and each start point of each signalsection, or the distance between each transition point included in eachof the plurality of signal sections and each end point of each signalsection exceeds the preset distance, the sensor device 110 may segmentthe signal section using the preset distance as a reference.

FIG. 6 is a flowchart of a method for segmenting, by the electronicdevice 10 (e.g., the electronic device 10 in FIG. 1 or the electronicdevice 1401 in FIG. 14) according to an embodiment, a sensor signalusing a crossing point and a transition point.

In an embodiment, the electronic device 10 may segment the sensor signalinto the plurality of signal sections using the crossing point, detectthe signal section including the plurality of transition points amongthe plurality of signal sections using the transition point, and segmentthe detected signal section such that the plurality of transition pointsare included in the different signal sections, respectively.

Referring to FIG. 6, according to an embodiment, in operation 610, thesensor device 110 may segment the sensor signal into the plurality ofsignal sections using the crossing point between the sensor signal andthe preset reference line (e.g., a line having a y value of 0) as areference. In an embodiment, when the number of crossing points betweenthe sensor signal and the preset reference line is equal to or less thanthe preset number, the sensor device 110 may change the value of thepreset reference line (e.g., operation 420 in FIG. 4).

According to an embodiment, in operation 620, for each of the pluralityof signal sections generated by being segmented in operation 610, thesensor device 110 may detect the signal section including the pluralityof transition points. For example, the sensor device 110 may sense thechange in the slope of the sensor signal, and detect the signal sectionincluding a plurality of points at which the slope changes from thepositive value to the negative value or from the negative value to thepositive value.

According to an embodiment, in operation 630, for the signal section(e.g., the signal section including the plurality of transition points)detected in operation 620, the sensor device 110 may segment the signalsection detected in operation 620 such that the plurality of transitionpoints are included in the different signal sections, respectively(e.g., operation 520 in FIG. 5).

According to an embodiment, in operation 640, the sensor device 110 maydetermine whether the amount of change in the x value or the y valuebetween the adjacent transition points among the plurality of transitionpoints is less than the preset amount of change (e.g., operation 530 inFIG. 5).

According to an embodiment, when it is determined in operation 640 thatthe amount of change in the x value or the y value between the adjacenttransition points is less than the preset amount of change (640-yes), inoperation 650, the sensor device 110 may allow the two transition pointswhose amount of change in the x value or amount of change in the y valueis less than the preset amount of change to be included in one signalsection (e.g., operation 540 in FIG. 5).

In an embodiment, the plurality of transition points include the firsttransition point and the second transition point set at the positionadjacent to the first transition point. The sensor device 110 segmentsthe sensor signal such that the first transition point and the secondtransition point are included in different signal sections. When amagnitude of the first transition point (e.g., an intensity of thesensor signal at the transition point) is equal to or greater than apreset magnitude and a magnitude of the second transition point is lessthan the preset magnitude, the sensor device 110 may include the firsttransition point and the second transition point in one signal section.

According to an embodiment, in operation 660, for each of the pluralityof signal sections, the sensor device 110 may determine whether thedistance between each transition point included in each of the pluralityof signal sections and each start point or each end point of thecorresponding signal section exceeds the preset distance (e.g.,operation 550 in FIG. 5).

According to an embodiment, in operation 670, for the signal section inwhich the distance between each transition point included in each of theplurality of signal sections and each start point or each end point ofthe corresponding signal section exceeds the preset distance, the sensordevice 110 may segment the signal section using the preset distance as areference (e.g., operation 560 in FIG. 5). For example, when thedistance between the first transition point included in the first signalsection and the start point of the first signal section exceeds thepreset distance (660-yes), the sensor device 110 may segment the firstsignal section using a point separated by the preset distance in adirection of the first transition point from the start point of thefirst signal section as a reference. In an embodiment, when both thedistance between the first transition point included in the first signalsection and the start point of the first signal section and the distancebetween the first transition point included in the first signal sectionand the end point of the first signal section exceed the presetdistance, the sensor device 110 may segment the first signal sectionsuch that the first transition point is included in a signal sectionhaving a width of a preset distance around the first transition point.

Conversely, when the distance between the first transition pointincluded in the first signal section and the start point of the firstsignal section does not exceed the preset distance (660-no), the sensordevice 110 may terminate the method for segmenting the sensor signalwithout segmenting the signal section using the preset distance as areference.

FIG. 7 is a diagram illustrating a form in which a sensor signal issegmented using a crossing point in an embodiment, and FIG. 8 is adiagram illustrating a form in which a sensor signal is segmented usinga crossing point and a transition point in an embodiment.

In FIG. 7, the signal is divided into sections 721, 722, and 723, ateach crossing point 711 . . . 714. However section 721 has threetransition points.

First, referring to FIG. 7, according to certain embodiments, theelectronic device 10 (e.g., the electronic device 10 in FIG. 1 or theelectronic device 1401 in FIG. 14) may set a crossing point 710 (e.g., afirst crossing point 711, a second crossing point 712, a third crossingpoint 713, and a fourth crossing point 714) using the sensor signal andthe preset reference line. The electronic device 10 may segment thesensor signal into the plurality of signal sections using the firstcrossing point 711, the second crossing point 712, the third crossingpoint 713, and the fourth crossing point 714. For example, theelectronic device 10 may primarily segment the sensor signal into afirst signal section 721 between the first crossing point 711 and thesecond crossing point 712, a second signal section 722 between thesecond crossing point 712 and the third crossing point 713, and a thirdsignal section 723 between the third crossing point 713 and the fourthcrossing point 714.

Referring to FIG. 8, section 721/821 which has three transition points831, 832, and 833 is further signal sections 821-1, 821-2, and 821-3.

The electronic device 10 may set the transition point from the sensorsignal, and detect a signal section including a plurality of transitionpoints from the primarily segmented sensor signal (e.g., the primarilysegmented sensor signal in FIG. 7) using the transition point. Forexample, the electronic device 10 may sense the change in the slope ofthe sensor signal (e.g., a degree to which the value of the sensorsignal changes during a unit time), and may detect a first signalsection 821 including points (e.g., a first transition point 831, asecond transition point 832, and a third transition point 833) at whichthe slope of the sensor signal changes from the positive value to thenegative value or from the negative value to the positive value amongthe first signal section 821, the second signal section 822, and thethird signal section 823. For example, the point at which the slope ofthe sensor signal changes from the positive value to the negative valuemay be a point at which a value of a gradually rising signal starts tofall again after reaching a maximum value. In an embodiment, theelectronic device 10 may secondarily segment the first signal section821 such that the first transition point 831, the second transitionpoint 832, and the third transition point 833 included in the firstsignal section 821 are respectively included in different signalsections (e.g., segment the first signal section 821 into a firstsection 821-1 of the first signal section, a second section 821-2 of thefirst signal section, and a third section 821-3 of the first signalsection).

FIG. 9 is a diagram illustrating a form of segmentation of a sensorsignal based on an amount of change in an x value or a y value betweenadjacent transition points in an embodiment.

In an embodiment, when the plurality of transition points are includedin one signal section, the electronic device 10 (e.g., the electronicdevice 10 in FIG. 1 or the electronic device 1401 in FIG. 14) maysegment the signal section such that the plurality of transition pointsare respectively included in the different signal sections, and maydetermine whether to segment the signal section based on the amount ofchange in the x value or the amount of change in the y value between theadjacent transition points among the plurality of transition points.

Referring to FIG. 9, the electronic device 10 may set the crossing point(e.g., a first crossing point 911, a second crossing point 912, a thirdcrossing point 913, and a fourth crossing point 914) and the transitionpoint (e.g., a first transition point 931, a second transition point932, a third transition point 933, a fourth transition point 934, and afifth transition point 935) from the sensor signal. The electronicdevice 10 may segment the sensor signal into a first signal section 921,a second signal section 922, and a third signal section 923 using thefirst crossing point 911, the second crossing point 912, the thirdcrossing point 913, and the fourth crossing point 914. The electronicdevice 10 may detect a signal section including a plurality oftransition points among the first signal section 921, the second signalsection 922, and the third signal section 923 using the first transitionpoint 931, the second transition point 932, the third transition point933, the fourth transition point 934, and the fifth transition point935.

The electronic device 10 may segment the signal section including theplurality of transition points such that the transition points areincluded in different signal sections, respectively. The electronicdevice 10 may segment the first signal section 921 such that the firsttransition point 931, the second transition point 932, and the thirdtransition point 933 are included in different signal sections, and maydetermine whether to segment the first signal section 921 based on anamount of change in an x value or a y value between the first transitionpoint 931 and the second transition point 932, and an amount of changein an x value or a y value between the second transition point 932 andthe third transition point 933. For example, when the amount of changein the y value (Δy) between the first transition point 931 and thesecond transition point 932 is less than the preset amount of change,the electronic device 10 may include the first transition point 931 andthe second transition point 932 in one signal section. When the amountof change in the y value (Δy) between the second transition point 932and the third transition point 933 is equal to or greater than thepreset amount of change, the electronic device 10 may segment the firstsignal section 921 such that the second transition point 932 and thethird transition point 933 are included in different signal sections.

FIG. 10 is a diagram illustrating a form of segmenting a sensor signalbased on a distance between a transition point and a start point or anend point of a signal section, and selecting a candidate section amongthe segmented sensor signals in an embodiment.

Referring to FIG. 10, the electronic device 10 (e.g., the electronicdevice 10 in FIG. 1 or the electronic device 1401 in FIG. 14) accordingto an embodiment may segment the sensor signal into a first signalsection 1021, a second signal section 1022, a third signal section 1023,a fourth signal section 1024, and a fifth signal section 1025 using thecrossing point between the sensor signal and the preset reference lineas a reference. The electronic device 10 may segment the sensor signalsuch that each of the first signal section 1021, the second signalsection 1022, the third signal section 1023, the fourth signal section1024, and the fifth signal section 1025 includes one transition pointusing a first transition point 1031, a second transition point 1032, athird transition point 1033, a fourth transition point 1034, and a fifthtransition point 1035 using the slope of the sensor signal.

In an embodiment, the electronic device 10 may determine whether each ofthe first signal section 1021, the second signal section 1022, the thirdsignal section 1023, the fourth signal section 1024, and the fifthsignal section 1025 exceeds a distance between a start point or an endpoint of each signal section and a transition point included in eachsignal section and a preset distance d₀ to detect a signal sectionexceeding the preset distance d₀. The electronic device 10 may segmentthe signal section exceeding the preset distance d₀ based on the presetdistance d₀. For example, when a distance d between the fourthtransition point 1034 included in the fourth signal section 1024 and anend point of the fourth signal section 1024 exceeds the preset distanced₀, the electronic device 10 may segment the fourth signal section 1024by the preset distance d₀ from the end point of the fourth signalsection 1024 (e.g., a first section 1024-1 of the fourth signal sectionand a second section 1024-2 of the fourth signal section).

In an embodiment, the electronic device 10 may determine whether atleast one transition point is included in each of the plurality ofsignal sections, and set signal sections, each of which including atleast one transition point among the plurality of signal sections as aplurality of candidate sections for recognizing the gesture of the user.For example, the electronic device 10 may set, among the plurality ofsignal sections, the first signal section 1021 including the firsttransition point 1031, the second signal section 1022 including thesecond transition point 1032, the third signal section 1023 includingthe third transition point 1033, the first section 1024-1 of the fourthsignal section 1024 including the fourth transition point 1034, and thefifth signal section 1025 including the fifth transition point 1035 asthe candidate sections.

FIG. 11 is a diagram illustrating a form of selecting candidate sectionsbased on an intensity of a sensor signal in an embodiment.

Referring to FIG. 11, the electronic device 10 (e.g., the electronicdevice 10 in FIG. 1 or the electronic device 1401 in FIG. 14) accordingto an embodiment may set the candidate sections for the gesturerecognition among the plurality of signal sections using an intensity(an amplitude) of a transition point included in each of the pluralityof signal sections. For example, the electronic device 10 may calculatean intensity A of the transition point included in each of the pluralityof signal sections, detect signal sections (e.g., a first signal section1121, a third signal section 1123, and a fourth signal section 1124) inwhich the calculated intensity of each transition point is equal to orabove a preset intensity A₀, and set the detected signal sections as theplurality of candidate sections for recognizing the gesture of the user.In an embodiment, the electronic device 10 may set signal sections inwhich an integral value of a plurality of sensor signals is less than apreset integral value among the plurality of signal sections as theplurality of candidate sections for recognizing the gesture of the user.

In an embodiment, the electronic device 10 may set a weighted value ofthe gesture recognition for each of the plurality of candidate sections,and may apply each weighted value to each of the plurality of candidatesections such that each weighted value is inversely proportional to theintensity of the sensor signal included in each of the plurality ofcandidate sections. For example, the electronic device 10 may set theweighted values for the first signal section 1121, the third signalsection 1123, and the fourth signal section 1124 set as the candidatesections such that the first signal section 1121 has the highestweighted value, the fourth signal section 1124 has the second highestweighted value, and the third signal section 1123 has the lowestweighted value. In an embodiment, the electronic device 10 may recognizethe gesture of the user using only a feature value extracted from acandidate section whose weighted value is equal to or greater than areference among feature values respectively extracted from the pluralityof candidate sections.

FIG. 12A and FIG. 12B are a diagram illustrating a heartbeat signal of auser through a frequency density function in an embodiment.

Referring to FIG. 12A and FIG. 12B, when the electronic device 10including the PPG sensor (e.g., the electronic device 10 in FIG. 1 orthe electronic device 1401 in FIG. 14) is intended to extract a featurevalue indicating the gesture of the user from a heartbeat signal sensedfrom the user, the electronic device 10 extracts a gesture signal fromthe heartbeat signal and analyzes the extracted gesture signal toextract the feature value. However, in a case in which the heartbeatsignal and the gesture signal are sensed together as shown in FIG. 12Bcompared to a case in which only the heartbeat signal is sensed as shownin FIG. 12A, because the gesture signal and the heartbeat are sensed ina state of being mixed with each other (because frequency domains of thegesture signal and the heartbeat signal overlap each other in terms of afrequency), even though the sensor signal is filtered through thepre-processing operation (e.g., a noise signal is removed from thesensor signal using the bandpass filter), there is a problem that it isdifficult to distinguish the gesture signal in terms of the frequency.In addition, when the electronic device 10 is the wearable device, astate (e.g., an age, a gender, a race, a skin color, a health status,and the like) of the user wearing the wearable device, a wearing state(e.g., a wearing intensity, a wearing position, and the like) may affectthe sensed sensor signal, and the waveform, a magnitude, and the like ofthe sensor signal may vary by an external environmental factor. That is,because the sensor signal sensed from the sensor device 110 is affectednot only by the state of the user and the wearing state, but also by theexternal factor, even when different users perform the same gesture,sensor signals of different waveforms and magnitudes are sensed, whichadversely affects the extraction of a common feature.

To solve the above problem, when the sensor signal includes a signalhaving a predetermined pattern (e.g., a bio-signal including a PPGsignal and a noise signal of a certain pattern) and the gesture signal,the electronic device 10 (e.g., the electronic device 10 in FIG. 1 orthe electronic device 1401 in FIG. 14) according to an embodiment mayextract the feature value for the gesture using information of a signalsection including the gesture signal (e.g., the maximum value, anaverage value, an energy value, the correlation, the entropy, the FastFourier Transform coefficient, the mean, the variance, the covariance,the time series data length, the skewness, kurtosis, and integral valuesof the sensor signal) and information of a signal section that does notinclude the gesture signal. For example, the electronic device 10 mayextract the feature value using also a signal before the gesture occurs.Information of a waveform before the gesture sensed from the sensordevice 110 occurs may include information reflecting the state of theuser and the external environmental factors. The electronic device 10may compensate for the sensor values that are different based on theenvironmental factor despite of being resulted from the same gesture andeffectively extract the feature value by mixing the information of thewaveform before the gesture occurs and information of a waveform of thegesture section with each other and using the mixed information as onefeature. An embodiment of this will be described in detail withreference to FIG. 13.

FIG. 13A and FIG. 13B illustrates a PPG signal (FIG. 13A) sensed from auser A and a PPG signal (FIG. 13B) sensed from a user B when differentusers (e.g., the user A and the user B) perform the same gesture.

Referring to FIGS. 13A and 13B, even when the different users performthe same gesture, the different waveforms may be sensed based on thestate of the user, the wearing state, and the external factor. While thePPG signal (FIG. 13A) sensed from the user A has a maximum value (apeak) 1331 of a heartbeat waveform of 397.9 in the signal section thatdoes not include the gesture signal, and a maximum value (a peak) 1332of 1639 in the signal section that includes the gesture signal, the PPGsignal (FIG. 13B) sensed from the user B has a maximum value (a peak)1333 of a heartbeat waveform of 503.7 in the signal section that doesnot include the gesture signal, and a maximum value (a peak) 1334 of2056 in the signal section that includes the gesture signal.

In a case of a conventional gesture recognition method (e.g., a methodof using the maximum value of the gesture signal as the feature value),even when the user A and the user B perform the same gesture operation,because the maximum values of the sensor signals respectively sensed bythe users are sensed differently, it is difficult for the user A and theuser B to extract the common feature. A description will be achievedusing Table 3 as an example.

TABLE 3 Maximum First value method Second method Person A 1639 4.441639/397 = 4.12 Person B 2056 3.79 2056/503 = 4.08

Referring to Table 3, when using a normalization technique, which is afirst method, on different values to solve this, feature valuesextracted from the user A and the user B are compensated to some extentto have values within a certain range (e.g., the feature value of theuser A is calculated to be 4.44, and the feature value of the user B iscalculated to be 3.79). However, because a range of the calculatedfeature value is somewhat large, it is difficult to accurately recognizethe gesture.

When using a second method of Table 3 according to an embodiment toovercome the problem in the conventional gesture recognition method, theelectronic device 10 may extract a ratio between a maximum value of asensor signal intensity in the signal section including the gesturesignal and a maximum value of a sensor signal intensity in a generalsignal section (e.g., the signal section that does not include thegesture signal) as the feature value for the gesture. For example, theelectronic device 10 may extract, from the sensor signal sensed from theuser A, 4.12, which is a ratio of 1639, which is the maximum value 1332of the signal intensity of the signal section including the gesturesignal, and 397.9, which is the maximum value 1331 of the signalintensity of the signal section that does not include the gesture signalas the feature value. The electronic device 10 may extract, from thesensor signal sensed from the user B, 4.08, which is a ratio of 2056,which is the maximum value 1334 of the signal intensity of the signalsection including the gesture signal, and 503.7, which is the maximumvalue 1333 of the signal intensity of the signal section that does notinclude the gesture signal as the feature value. In an embodiment, theelectronic device 10 may perform an operation (e.g., various statisticaloperations including four arithmetic operations, comparison operations,logical operations, spectrum analysis, correlation coefficient,variance, standard deviation operations, and the like) betweeninformation of the signal section including the gesture signal andinformation of the signal section that does not include the gesturesignal, and extract the result of the operation as the feature value forthe gesture.

FIG. 14 is a block diagram of an electronic device 1401 in a networkenvironment 1400 according to certain embodiments. Referring to FIG. 14,the electronic device 1401 may communicate with an electronic device1402 through a first network 1498 (e.g., a short-range wirelesscommunication network) or may communicate with an electronic device 1404or a server 1408 through a second network 1499 (e.g., a long-distancewireless communication network) in the network environment 1400.According to an embodiment, the electronic device 1401 may communicatewith the electronic device 1404 through the server 1408. According to anembodiment, the electronic device 1401 may include a processor 1420, amemory 1430, an input device 1450, a sound output device 1455, a displaydevice 1460, an audio module 1470, a sensor module 1476, an interface1477, a haptic module 1479, a camera module 1480, a power managementmodule 1488, a battery 1489, a communication module 1490, a subscriberidentification module 1496, or an antenna module 1497. According to someembodiments, at least one (e.g., the display device 1460 or the cameramodule 1480) among components of the electronic device 1401 may beomitted or one or more other components may be added to the electronicdevice 1401. According to some embodiments, some of the above componentsmay be implemented with one integrated circuit. For example, the sensormodule 1476 (e.g., a fingerprint sensor, an iris sensor, or anilluminance sensor) may be embedded in the display device 1460 (e.g., adisplay).

The processor 1420 may execute, for example, software (e.g., a program1440) to control at least one of other components (e.g., a hardware orsoftware component) of the electronic device 1401 connected to theprocessor 1420 and may process or compute a variety of data. Accordingto an embodiment, as a part of data processing or operation, theprocessor 1420 may load a command set or data, which is received fromother components (e.g., the sensor module 1476 or the communicationmodule 1490), into a volatile memory 1432, may process the command ordata loaded into the volatile memory 1432, and may store result datainto a nonvolatile memory 1434. According to an embodiment, theprocessor 1420 may include a main processor 1421 (e.g., a centralprocessing unit or an application processor) and an auxiliary processor1423 (e.g., a graphic processing device, an image signal processor, asensor hub processor, or a communication processor), which operatesindependently from the main processor 1421 or with the main processor1421. Additionally or alternatively, the auxiliary processor 1423 mayuse less power than the main processor 1421, or is specified to adesignated function. The auxiliary processor 1423 may be implementedseparately from the main processor 1421 or as a part thereof.

The auxiliary processor 1423 may control, for example, at least some offunctions or states associated with at least one component (e.g., thedisplay device 1460, the sensor module 1476, or the communication module1490) among the components of the electronic device 1401 instead of themain processor 1421 while the main processor 1421 is in an inactive(e.g., sleep) state or together with the main processor 1421 while themain processor 1421 is in an active (e.g., an application execution)state. According to an embodiment, the auxiliary processor 1423 (e.g.,the image signal processor or the communication processor) may beimplemented as a part of another component (e.g., the camera module 1480or the communication module 1490) that is functionally related to theauxiliary processor 1423.

The memory 1430 may store a variety of data used by at least onecomponent (e.g., the processor 1420 or the sensor module 1476) of theelectronic device 1401. For example, data may include software (e.g.,the program 1440) and input data or output data with respect to commandsassociated with the software. The memory 1430 may include the volatilememory 1432 or the nonvolatile memory 1434.

The program 1440 may be stored in the memory 1430 as software and mayinclude, for example, an operating system 1442, a middleware 1444, or anapplication 1446.

The input device 1450 may receive a command or data, which is used for acomponent (e.g., the processor 1420) of the electronic device 1401, froman outside (e.g., a user) of the electronic device 1401. The inputdevice 1450 may include, for example, a microphone, a mouse, a keyboard,or a digital pen (e.g., a stylus pen).

The sound output device 1455 may output a sound signal to the outside ofthe electronic device 1401. The sound output device 1455 may include,for example, a speaker or a receiver. The speaker may be used forgeneral purposes, such as multimedia play or recordings play, and thereceiver may be used for receiving calls. According to an embodiment,the receiver and the speaker may be either integrally or separatelyimplemented.

The display device 1460 may visually provide information to the outside(e.g., the user) of the electronic device 1401. For example, the displaydevice 1460 may include a display, a hologram device, or a projector anda control circuit for controlling a corresponding device. According toan embodiment, the display device 1460 may include a touch circuitryconfigured to sense the touch or a sensor circuit (e.g., a pressuresensor) for measuring an intensity of pressure on the touch.

The audio module 1470 may convert a sound and an electrical signal indual directions. According to an embodiment, the audio module 1470 mayobtain the sound through the input device 1450 or may output the soundthrough the sound output device 1455 or an external electronic device(e.g., the electronic device 1402 (e.g., a speaker or a headphone))directly or wirelessly connected to the electronic device 1401.

The sensor module 1476 may generate an electrical signal or a data valuecorresponding to an operating state (e.g., power or temperature) insideor an environmental state (e.g., a user state) outside the electronicdevice 1401. According to an embodiment, the sensor module 1476 mayinclude, for example, a gesture sensor, a gyro sensor, a barometricpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 1477 may support one or more designated protocols to allowthe electronic device 1401 to connect directly or wirelessly to theexternal electronic device (e.g., the electronic device 1402). Accordingto an embodiment, the interface 1477 may include, for example, an HDMI(high-definition multimedia interface), a USB (universal serial bus)interface, an SD card interface, or an audio interface.

A connecting terminal 1478 may include a connector that physicallyconnects the electronic device 1401 to the external electronic device(e.g., the electronic device 1402). According to an embodiment, theconnecting terminal 1478 may include, for example, an HDMI connector, aUSB connector, an SD card connector, or an audio connector (e.g., aheadphone connector).

The haptic module 1479 may convert an electrical signal to a mechanicalstimulation (e.g., vibration or movement) or an electrical stimulationperceived by the user through tactile or kinesthetic sensations.According to an embodiment, the haptic module 1479 may include, forexample, a motor, a piezoelectric element, or an electric stimulator.

The camera module 1480 may shoot a still image or a video image.According to an embodiment, the camera module 1480 may include, forexample, at least one or more lenses, image sensors, image signalprocessors, or flashes.

The power management module 1488 may manage power supplied to theelectronic device 1401. According to an embodiment, the power managementmodule 1488 may be implemented as at least a part of a power managementintegrated circuit (PMIC).

The battery 1489 may supply power to at least one component of theelectronic device 1401. According to an embodiment, the battery 1489 mayinclude, for example, a non-rechargeable (primary) battery, arechargeable (secondary) battery, or a fuel cell.

The communication module 1490 may establish a direct (e.g., wired) orwireless communication channel between the electronic device 1401 andthe external electronic device (e.g., the electronic device 1402, theelectronic device 1404, or the server 1408) and support communicationexecution through the established communication channel. Thecommunication module 1490 may include at least one communicationprocessor operating independently from the processor 1420 (e.g., theapplication processor) and supporting the direct (e.g., wired)communication or the wireless communication. According to an embodiment,the communication module 1490 may include a wireless communicationmodule 1492 (e.g., a cellular communication module, a short-rangewireless communication module, or a GNSS (global navigation satellitesystem) communication module) or a wired communication module 1494(e.g., an LAN (local area network) communication module or a power linecommunication module). The corresponding communication module among theabove communication modules may communicate with the external electronicdevice 1404 through the first network 1498 (e.g., the short-rangecommunication network such as a Bluetooth, a WiFi direct, or an IrDA(infrared data association)) or the second network 1499 (e.g., thelong-distance wireless communication network such as a cellular network,an internet, or a computer network (e.g., LAN or WAN)). Theabove-mentioned various communication modules may be implemented intoone component (e.g., a single chip) or into separate components (e.g.,chips), respectively. The wireless communication module 1492 mayidentify and authenticate the electronic device 1401 using userinformation (e.g., international mobile subscriber identity (IMSI))stored in the subscriber identification module 1496 in the communicationnetwork, such as the first network 1498 or the second network 1499.

The antenna module 1497 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 1401. According to an embodiment, the antenna module1497 may include an antenna including a radiating element composed of aconductive material or a conductive pattern formed in or on a substrate(e.g., PCB). According to an embodiment, the antenna module 1497 mayinclude a plurality of antennas. In such a case, at least one antennaappropriate for a communication scheme used in the communicationnetwork, such as the first network 1498 or the second network 1499, maybe selected, for example, by the communication module 1490 from theplurality of antennas. The signal or the power may then be transmittedor received between the communication module 1490 and the externalelectronic device via the selected at least one antenna. According to anembodiment, another component (e.g., a radio frequency integratedcircuit (RFIC)) other than the radiating element may be additionallyformed as part of the antenna module 1497.

At least some components among the components may be connected to eachother through a communication method (e.g., a bus, a GPIO (generalpurpose input and output), an SPI (serial peripheral interface), or anMIPI (mobile industry processor interface)) used between peripheraldevices to exchange signals (e.g., a command or data) with each other.

According to an embodiment, the command or data may be transmitted orreceived between the electronic device 1401 and the external electronicdevice 1404 through the server 1408 connected to the second network1499. Each of the external electronic devices 1402 and 1404 may be thesame or different types as or from the electronic device 1401. Accordingto an embodiment, all or some of the operations performed by theelectronic device 1401 may be performed by one or more externalelectronic devices among the external electronic devices 1402, 1404, or1408. For example, when the electronic device 1401 performs somefunctions or services automatically or by request from a user or anotherdevice, the electronic device 1401 may request one or more externalelectronic devices to perform at least some of the functions related tothe functions or services, in addition to or instead of performing thefunctions or services by itself. The one or more external electronicdevices receiving the request may carry out at least a part of therequested function or service or the additional function or serviceassociated with the request and transmit the execution result to theelectronic device 1401. The electronic device 1401 may provide theresult as is or after additional processing as at least a part of theresponse to the request. To this end, for example, a cloud computing,distributed computing, or client-server computing technology may beused.

The electronic device according to certain embodiments disclosed in thedisclosure may be various types of devices. The electronic device mayinclude, for example, a portable communication device (e.g., asmartphone), a computer device, a portable multimedia device, a mobilemedical appliance, a camera, a wearable device, or a home appliance. Theelectronic device according to an embodiment of the disclosure shouldnot be limited to the above-mentioned devices.

It should be understood that certain embodiments of the disclosure andterms used in the embodiments do not intend to limit technical featuresdisclosed in the disclosure to the particular embodiment disclosedherein; rather, the disclosure should be construed to cover variousmodifications, equivalents, or alternatives of embodiments of thedisclosure. With regard to description of drawings, similar or relatedcomponents may be assigned with similar reference numerals. As usedherein, singular forms of noun corresponding to an item may include oneor more items unless the context clearly indicates otherwise. In thedisclosure disclosed herein, each of the expressions “A or B”, “at leastone of A and B”, “at least one of A or B”, “A, B, or C”, “one or more ofA, B, and C”, or “one or more of A, B, or C”, and the like used hereinmay include any and all combinations of one or more of the associatedlisted items. The expressions, such as “a first”, “a second”, “thefirst”, or “the second”, may be used merely for the purpose ofdistinguishing a component from the other components, but do not limitthe corresponding components in other aspect (e.g., the importance orthe order). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

The term “module” used in the disclosure may include a unit implementedin hardware, software, or firmware and may be interchangeably used withthe terms “logic”, “logical block”, “part” and “circuit”. The “module”may be a minimum unit of an integrated part or may be a part thereof.The “module” may be a minimum unit for performing one or more functionsor a part thereof. For example, according to an embodiment, the “module”may include an application-specific integrated circuit (ASIC).

Certain embodiments of the disclosure may be implemented by software(e.g., the program 1440) including an instruction stored in amachine-readable storage medium (e.g., an internal memory 1436 or anexternal memory 1438) readable by a machine (e.g., the electronic device1401). For example, the processor (e.g., the processor 1420) of amachine (e.g., the electronic device 1401) may call the instruction fromthe machine-readable storage medium and execute the instructions thuscalled. This means that the machine may perform at least one functionbased on the called at least one instruction. The one or moreinstructions may include a code generated by a compiler or executable byan interpreter. The machine-readable storage medium may be provided inthe form of non-transitory storage medium. Here, the term“non-transitory”, as used herein, means that the storage medium istangible, but does not include a signal (e.g., an electromagnetic wave).The term “non-transitory” does not differentiate a case where the datais permanently stored in the storage medium from a case where the datais temporally stored in the storage medium.

According to an embodiment, the method according to certain embodimentsdisclosed in the disclosure may be provided as a part of a computerprogram product. The computer program product may be traded between aseller and a buyer as a product. The computer program product may bedistributed in the form of machine-readable storage medium (e.g., acompact disc read only memory (CD-ROM)) or may be directly distributed(e.g., download or upload) online through an application store (e.g., aPlay Store™) or between two user devices (e.g., the smartphones). In thecase of online distribution, at least a portion of the computer programproduct may be temporarily stored or generated in a machine-readablestorage medium such as a memory of a manufacturer's server, anapplication store's server, or a relay server.

According to certain embodiments, each component (e.g., the module orthe program) of the above-described components may include one or pluralentities. According to certain embodiments, at least one or morecomponents of the above components or operations may be omitted, or oneor more components or operations may be added. Alternatively oradditionally, some components (e.g., the module or the program) may beintegrated in one component. In this case, the integrated component mayperform the same or similar functions performed by each correspondingcomponents prior to the integration. According to certain embodiments,operations performed by a module, a programming, or other components maybe executed sequentially, in parallel, repeatedly, or in a heuristicmethod, or at least some operations may be executed in differentsequences, omitted, or other operations may be added.

Certain embodiments of the disclosure and terms used herein are notintended to limit the technologies described in the disclosure tospecific embodiments, and it should be understood that the embodimentsand the terms include modification, equivalent, and/or alternative onthe corresponding embodiments described herein. With regard todescription of drawings, similar components may be marked by similarreference numerals. The terms of a singular form may include pluralforms unless otherwise specified. In the disclosure disclosed herein,the expressions “A or B”, “at least one of A and/or B”, “A, B, or C”, or“at least one of A, B, and/or C”, and the like used herein may includeany and all combinations of one or more of the associated listed items.Expressions such as “first,” or “second,” and the like, may expresstheir components regardless of their priority or importance and may beused to distinguish one component from another component but is notlimited to these components. When an (e.g., first) component is referredto as being “(operatively or communicatively) coupled with/to” or“connected to” another (e.g., second) component, it may be directlycoupled with/to or connected to the other component or an interveningcomponent (e.g., a third component) may be present.

According to the situation, the expression “adapted to or configured to”used herein may be interchangeably used as, for example, the expression“suitable for”, “having the capacity to”, “changed to”, “made to”,“capable of” or “designed to” in hardware or software. The expression “adevice configured to” may mean that the device is “capable of” operatingtogether with another device or other parts. For example, a “processorconfigured to (or set to) perform A, B, and C” may mean a dedicatedprocessor (e.g., an embedded processor) for performing correspondingoperations or a generic-purpose processor (e.g., a central processingunit (CPU) or an application processor (AP)) which performscorresponding operations by executing one or more software programswhich are stored in a memory device (e.g., the memory 1430).

The term “module” used herein may include a unit, which is implementedwith hardware, software, or firmware, and may be interchangeably usedwith the terms “logic”, “logical block”, “part”, “circuit”, or the like.The “module” may be a minimum unit of an integrated part or a partthereof or may be a minimum unit for performing one or more functions ora part thereof. The “module” may be implemented mechanically orelectronically and may include, for example, an application-specific IC(ASIC) chip, a field-programmable gate array (FPGA), and aprogrammable-logic device for performing some operations, which areknown or will be developed.

At least a part of an apparatus (e.g., modules or functions thereof) ora method (e.g., operations) according to certain embodiments may be, forexample, implemented by instructions stored in a computer-readablestorage media (e.g., the memory 1430) in the form of a program module.The instruction, when executed by a processor (e.g., the processor1420), may cause the processor to perform a function corresponding tothe instruction. The computer-readable recording medium may include ahard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), anoptical media (e.g., a compact disc read only memory (CD-ROM) and adigital versatile disc (DVD), a magneto-optical media (e.g., a flopticaldisk)), an embedded memory, and the like. The one or more instructionsmay contain a code made by a compiler or a code executable by aninterpreter.

Each component (e.g., a module or a program module) according to certainembodiments may be composed of single entity or a plurality of entities,a part of the above-described sub-components may be omitted, or othersub-components may be further included. Alternatively or additionally,after being integrated in one entity, some components (e.g., a module ora program module) may identically or similarly perform the functionexecuted by each corresponding component before integration. Accordingto certain embodiments, operations executed by modules, program modules,or other components may be executed by a successive method, a parallelmethod, a repeated method, or a heuristic method, or at least one partof operations may be executed in different sequences or omitted.Alternatively, other operations may be added.

According to certain embodiments disclosed in the disclosure, as thesensor signal sensed from the user is elaborately subdivided and thesubdivided signal sections are respectively analyzed to recognize thegesture, a result value of high reliability may be derived with a smallamount of operations for the gesture recognition.

According to certain embodiments disclosed in the disclosure, as anormal signal generally sensed from the user and the gesture signalsensed during the gesture operation are compared to each other toextract the feature value for the gesture, the gesture may be accuratelyrecognized from the different signals sensed by the different users.

In addition, various effects that are directly or indirectly identifiedthrough the disclosure may be provided.

While the disclosure has been shown and described with reference tocertain embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An electronic device comprising: a sensor deviceincluding at least one biometric sensor; memory storing a table of aplurality of gestures, wherein for each gesture, a correspondingplurality of features are stored in the memory; and at least oneprocessor operatively connected to the sensor device and the memory,wherein the at least one processor is configured to: obtain a bio-signalof a user from the at least one biometric sensor; select a section ofthe bio-signal that includes one feature of the corresponding pluralityof features for the plurality of gestures; determine a specific one ofthe plurality of gestures based on the one feature included in thesection of the bio-signal and the corresponding plurality features forthe plurality of gestures.
 2. The electronic device of claim 1, whereinthe processor is configured to: identify a crossing point between thebio-signal and a specified reference line and a transition point of thebio-signal; and select the section based on the crossing point and thetransition point.
 3. The electronic device of claim 2, wherein thespecified reference line is a line connecting points where a raw valuebecomes 0 with each other when a value of the bio-signal is the rawvalue.
 4. The electronic device of claim 2, wherein the specifiedreference line is a line connecting points where a value of thebio-signal normalized by the processor becomes 0 with each other whenthe value of the bio-signal is not a raw value.
 5. The electronic deviceof claim 2, wherein the sensor device is configured to segment thebio-signal using at least one of the crossing point and the transitionpoint, wherein the processor is configured to: extract a feature valuefrom the segmented sensor signals transmitted from the sensor device,and recognize the gesture using the extracted feature value and thecorresponding plurality features for the plurality of gestures.
 6. Theelectronic device of claim 2, wherein the sensor device is configuredto: segment the bio-signal into a plurality of signal sections using thecrossing point; detect a signal section including a plurality oftransition points among the plurality of signal sections using thetransition point; and segment the detected signal section such that theplurality of transition points are respectively included in differentsignal sections.
 7. The electronic device of claim 6, wherein theplurality of transition points include a first transition point and asecond transition point set at a position adjacent to the firsttransition point, wherein the sensor device is configured to segment thedetected signal section such that the first transition point and thesecond transition point are included in one signal section when at leastone of an amount of change in an x value and an amount of change in a yvalue between the first transition point and the second transition pointis less than a preset amount of change.
 8. The electronic device ofclaim 6, wherein the plurality of transition points include a firsttransition point and a second transition point set at a positionadjacent to the first transition point, wherein the sensor device isconfigured to segment the detected signal section such that the firsttransition point and the second transition point are included in onesignal section when a magnitude of the first transition point is equalto or greater than a preset magnitude, and a magnitude of the secondtransition point is less than the preset magnitude.
 9. The electronicdevice of claim 6, wherein the sensor device is configured to segment afirst signal section using a preset distance as a reference when, in thefirst signal section including a first transition point among theplurality of signal sections, a distance between the first transitionpoint and a start point of the first signal section or between the firsttransition point and an end point of the first signal section exceedsthe preset distance.
 10. The electronic device of claim 6, wherein theprocessor is configured to: determine whether at least one transitionpoint is included in each of the plurality of signal sections; and setsignal sections including the at least one transition point among theplurality of signal sections as a plurality of candidate sections forrecognizing the gesture of the user.
 11. The electronic device of claim10, wherein the processor is configured to apply each weighted value ofgesture recognition to each of the plurality of candidate sections,wherein each weighted value is inversely proportional to an intensity ofthe sensor signal for each candidate section.
 12. The electronic deviceof claim 6, wherein the processor is configured to set signal sectionswhere an intensity of the sensor signal is equal to or greater than apreset intensity among the plurality of signal sections as a pluralityof candidate sections.
 13. The electronic device of claim 6, wherein theprocessor is configured to set signal sections where an integral valueof the sensor signal is less than a preset integral value among theplurality of signal sections as a plurality of candidate sections. 14.The electronic device of claim 6, wherein the sensor device isconfigured to: segment the sensor signal into the plurality of signalsections using the crossing point; and set a preset reference line suchthat at least one transition point is included in each of the pluralityof signal sections.
 15. The electronic device of claim 6, wherein thesensor device is configured to change a value of the reference line suchthat a number of plurality of signal sections is equal to or greaterthan a preset number when the number of plurality of signal sections isless than the preset number.
 16. The electronic device of claim 1,wherein the sensor device includes at least one of an accelerationsensor, a photoplethysmography (PPG) sensor, a gyro sensor, ageomagnetic sensor, and an electrocardiogram sensor.
 17. The electronicdevice of claim 5, wherein the processor is configured to: define a formof each of the segmented sensor signals using the transition pointincluded in each of the segmented sensor signals; set a sequence of thesegmented sensor signals based on the form of each of the segmentedsensor signals; and extract the sequence of the segmented sensor signalsas the feature value.
 18. The electronic device of claim 5, wherein theprocessor is configured to independently extract the feature value fromeach of the segmented sensor signals.
 19. The electronic device of claim5, wherein the processor is configured to extract the feature valueusing information of a second signal section including a gesture signalof the segmented sensor signals and information of a third signalsection not including the gesture signal when the sensor signal includesa signal of a predetermined pattern and the gesture signal.
 20. A methodfor recognizing a gesture performed by an electronic device, the methodcomprising: sensing a sensor signal from a user; storing a table of aplurality of gestures, wherein for each gesture, a correspondingplurality of features are stored; setting a crossing point between thesensor signal and a preset reference line and a transition point of thesensor signal; segmenting the sensor signal using at least one of thecrossing point and the transition point; selecting a section of thesensor signal that includes one feature of the corresponding pluralityof features for the plurality of gestures; and determining a specificone of the plurality of gestures based on the one feature included inthe segment and the corresponding plurality features for the pluralityof gestures.